{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.core.display import display, HTML\n",
    "display(HTML(\"<style>.container { width:90% !important; }</style>\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lecture 9 -  Naive Bayes Classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Table of Contents\n",
    "* [Lecture 9 -  Naive Bayes Classifier](#Lecture-9----Naive-Bayes-Classifier)\n",
    "\t* &nbsp;\n",
    "\t\t* [Content](#Content)\n",
    "\t\t* [Learning Outcomes](#Learning-Outcomes)\n",
    "\t* &nbsp;\n",
    "\t\t* [Example dataset - Wine](#Example-dataset---Wine)\n",
    "* [Naive Bayes](#Naive-Bayes)\n",
    "\t* &nbsp;\n",
    "\t\t* &nbsp;\n",
    "\t\t\t* [Probabilities](#Probabilities)\n",
    "\t\t\t* [Joint probabilities](#Joint-probabilities)\n",
    "\t\t* [Probability density functions](#Probability-density-functions)\n",
    "\t\t* [Steps for calculating the classification for naive Bayes](#Steps-for-calculating-the-classification-for-naive-Bayes)\n",
    "\t\t* [Comparing Naive Bayes to k-NN](#Comparing-Naive-Bayes-to-k-NN)\n",
    "* [Putting it all together in scikit-learn](#Putting-it-all-together-in-scikit-learn)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Naive Bayes\n",
    "2. Scikit-learn and Naive Bayes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learning Outcomes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At the end of this lecture, you should be able to:\n",
    "\n",
    "* explain how Naive Bayes works\n",
    "* implement Naive Bayes in python\n",
    "* apply Naive Bayes to classification\n",
    "* use the scikit-learn module to train and test Naive Bayes classifiers\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example dataset - Wine"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will return to the Wine dataset to explore classification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib as mtpl\n",
    "import seaborn as sns\n",
    "\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "df = pd.io.parsers.read_csv(\n",
    "    '../datasets/wine_data.csv',\n",
    "     usecols=[0,6,7]\n",
    "    )\n",
    "\n",
    "df.columns=['Class','Magnesium','Flavanoids']\n",
    "\n",
    "df['Class'].replace('3', 0, inplace=True)\n",
    "df.to_csv('../datasets/wine_data_test.csv', header=None, index=None)\n",
    "\n",
    "df.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Confirm we have 3 class labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.Class.unique()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Confirm data types:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get counts for each class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.groupby('Class').count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.groupby('Class').count() / df.groupby('Class').count()['Magnesium'].sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In classification problems, the **ability to separate classes** from one another is the most important consideration. Histograms of the feature values per class type, can be a useful tool for **eyeballing** some features and to get a rough feeling for their **discriminative power**. \n",
    "\n",
    "Here we are visualising the histograms of the two features for each of the three classes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "plt.figure(figsize=(10,8))\n",
    "\n",
    "colors = ('blue', 'red', 'green')\n",
    "\n",
    "for label,color in zip(range(1,4), colors):\n",
    "    mean = np.mean(df['Magnesium'][df['Class'] == label]) # class sample mean\n",
    "    stdev = np.std(df['Magnesium'][df['Class'] == label]) # class standard deviation\n",
    "    df['Magnesium'][df['Class'] == label].hist(alpha=0.3, # opacity level\n",
    "             label='class {} ($\\mu={:.2f}$, $\\sigma={:.2f}$)'.format(label, mean, stdev), \n",
    "             color=color,\n",
    "             bins=15)\n",
    "\n",
    "plt.title('Wine data set - Distribution of Magnesium content')\n",
    "plt.xlabel('Magnesium content', fontsize=14)\n",
    "plt.ylabel('count', fontsize=14)\n",
    "plt.legend(loc='upper right')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "\n",
    "plt.figure(figsize=(10,8))\n",
    "\n",
    "colors = ('blue', 'red', 'green')\n",
    "\n",
    "for label,color in zip(range(1,4), colors):\n",
    "    mean = np.mean(df['Flavanoids'][df['Class'] == label]) # class sample mean\n",
    "    stdev = np.std(df['Flavanoids'][df['Class'] == label]) # class standard deviation\n",
    "    df['Flavanoids'][df['Class'] == label].hist(alpha=0.3, # opacity level\n",
    "             label='class {} ($\\mu={:.2f}$, $\\sigma={:.2f}$)'.format(label, mean, stdev), \n",
    "             color=color,\n",
    "             bins=15)\n",
    "\n",
    "plt.title('Wine data set - Distribution of Flavanoids content')\n",
    "plt.xlabel('Flavanoids content', fontsize=14)\n",
    "plt.ylabel('count', fontsize=14)\n",
    "plt.legend(loc='upper right')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Naive Bayes - Probability of the Causes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Naive Bayes (NB) is a classical statistical machine learning algorithm. It is based on one of the most important equations both in statistics as well as in science as whole - the **Bayes Theorem**. \n",
    "\n",
    "The Bayes Theorem is the foundation of a large branch of statistics that has an increasing relevance in solving real-world data science problems called **Bayesian Statistics**. Under this subfield of statistics, advanced methods like, Bayesian networks, Hidden Markov Models, Markov Random fields and probabilistic relational models are situated.\n",
    "\n",
    "With the Bayes Theorem the **evidence about the true state of the world is expressed in terms of degrees of belief (probabilities)**. As such, the Bayes Theorem deals with **conditional probabilities between different events**. \n",
    "\n",
    "*It allows us to calculate the probability of some even A occurring given that some evidence B is true.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Probabilities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Say we are developing an email spam filter. We are dealing with two classes, **spam** and **non-spam**. We look through all the spam emails in our email account and we find that 20% of them have the word 'Viagra' in them, while 80% contain the word 'Bank'. We would express this as: \n",
    "\n",
    "\\begin{equation}\n",
    "p(Viagra) = 0.2\n",
    "\\end{equation}\n",
    "\n",
    "and\n",
    "\n",
    "\\begin{equation}\n",
    "p(Bank) = 0.8\n",
    "\\end{equation}\n",
    "\n",
    "If we assume that the occurrence of the above words in the emails is independent of each other (which means that one does not influence the other), than we can calculate the probability of encountering a spam email having both these words is:\n",
    "\n",
    "0.2 $\\times$ 0.8 = 0.16, which is p(Viagra)p(Bank).\n",
    "\n",
    "This can be fully expressed as:\n",
    "\n",
    "\\begin{equation}\n",
    "p(Viagra\\ and\\ Bank | Spam)\n",
    "\\end{equation}\n",
    "\n",
    "meaning the probability of Viagra and Bank occurring together, given a spam email.\n",
    "\n",
    "The above in effect expresses:\n",
    "\n",
    "\\begin{equation}\n",
    "p(Words | Class)\n",
    "\\end{equation}\n",
    "\n",
    "meaning the probability of certain words occurring for a given class which could be spam or non-spam. However, we are interested in prediction, so for us the above needs to be inverted because we want to know what is the probability of something being spam or non-spam given certain words we have come across in an email. In effect, what we are interested in is:\n",
    "\n",
    "\\begin{equation}\n",
    "p(Class | Words)\n",
    "\\end{equation}\n",
    "\n",
    "Our problem is that \n",
    "\n",
    "\\begin{equation}\n",
    "p(Class | Words) \\neq p(Words | Class)\n",
    "\\end{equation}\n",
    "\n",
    "\n",
    "Fortunately, 250 years ago, the British mathematician Reverend **Thomas Bayes** first proposed how to do this through a thought experiment. The great French mathematician, **Pierre Laplace**, also figured this out at independently and gave it mathematical formalism which we have today."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Joint probabilities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since\n",
    "\n",
    "\\begin{equation}\n",
    "p(A\\ and\\ B) = p(B)p(A) \n",
    "\\end{equation}\n",
    "\n",
    "**is not always true**, generally the joint probability of two events is therefore expressed as follows:\n",
    "\n",
    "\\begin{equation}\n",
    "p(A\\ and\\ B) = p(B|A)p(A)\n",
    "\\end{equation}\n",
    "\n",
    "where p(B|A) represents the probability of B occurring, given that A has occurred. The above equation is also  interchangeable as:\n",
    "\n",
    "\\begin{equation}\n",
    "p(A\\ and\\ B) = p(A|B)p(B)\n",
    "\\end{equation}\n",
    "\n",
    "by pulling the two pieces together we have: \n",
    "\n",
    "\\begin{equation}\n",
    "p(B|A)p(A) = p(A|B)p(B)\n",
    "\\end{equation}\n",
    "\n",
    "This now brings us to the Bayes Theorem and to the solution to our problem. For our problem, if we make B the class label of 'spam' (which is the probability we would like to predict), and A the words in the email (like viagra or bank), then we can solve by re-arranging this equation as:\n",
    "\n",
    "\\begin{equation}\n",
    "P(B|A) = \\frac{P(A | B)\\, P(B)}{P(A)}\\cdot\n",
    "\\end{equation}\n",
    "\n",
    "By plugging in our problem into the formula, we would get:\n",
    "\n",
    "\\begin{equation}\n",
    "P(Class|Words) = \\frac{P(Words | Class)\\, P(Class)}{P(Words)}\\cdot\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Breaking the formula down, P(Words|Class) is called the **likelihood** and we can calculate this from our training set. The likelihood can be phrased as given all the spam emails, what is the probability that the words like 'viagra' and 'bank' occur in any document. It is essentially the product of each of the probabilities for every individual word.\n",
    "\n",
    "Naive Bayes is naive, because it assumes independence amongst the features in the calculation of the likelihood which in our example we express simply as p(Viagra)p(Bank).\n",
    "\n",
    "The P(Class) is called the **prior**. This we can also calculate from our dataset as being the proportion of emails that are classed as spam.\n",
    "\n",
    "The P(Words) is called the **normalising constant** (which is simply the probability of seeing this pattern without knowing what class it belongs to  - this is the least important component and in the end reduces down to simply the sum of all class probabilities), while the P(Class|Words) is called the **posterior** and is the result we are looking for.\n",
    "\n",
    "Let us continue the email spam filter example, but before we proceed we will need to complete the describing our dataset.\n",
    "\n",
    "Say we were examining a total of 100 emails, both spam and non-spam. 40 of those were spam and the rest were not. Of the 60 emails that were not spam, the word 'Viagra' appeared 5% and the word 'Bank' 10%. Across the entire dataset, the probability of finding 'Viagra' or 'Bank' in the emails, irrespective of what class the email belonged to was 7% (but this part can be left out).\n",
    "\n",
    "Our goal is to find out which class label is the most probable given the particular word features:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#caclulate p(non-spam| viagra and bank)\n",
    "((0.05 * 0.1) * 0.6) / (0.07)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#caclulate p(spam| viagra and bank)\n",
    "((0.2 * 0.8) * 0.4) / (0.07)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Usually we use the normalising constant as below: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('non-spam: ', (0.05 * 0.1) * 0.6 / ((0.05 * 0.1) * 0.6 + (0.2 * 0.8) * 0.4) )\n",
    "print('spam: ', (0.2 * 0.8) * 0.4 / ((0.05 * 0.1) * 0.6 + (0.2 * 0.8) * 0.4) )\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From this we can say that the probability of an email containing words viagra and bank is more probable to be spam than non-spam and we therefore assign the class label to the feature ['Viagra','Bank'] as **spam**.\n",
    "\n",
    "In many classification tasks, you have to deal with incomplete or missing values. As it turns out **Naive Bayes is really good for dealing with missing values and is able to produce a classification without having all the features, whereby the likelihood is simply calculated by excluding the particular missing feature.**\n",
    "\n",
    "The above simplistic example considered features that were **categorical** and were represented as frequencies. \n",
    "\n",
    "How would we apply Naive Bayes to the wine dataset where the features for 'Magnesium' and 'Flavanoids' are numerical?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Probability density functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below is an example of the Gaussian (normal) probability density function (PDF)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import scipy.stats as stats\n",
    "\n",
    "h = np.random.randn(1000)\n",
    "h.sort()\n",
    "fit = stats.norm.pdf(h, np.mean(h), np.std(h))  #this is a fitting indeed\n",
    "\n",
    "plt.figure(figsize=(10,8))\n",
    "plt.plot(h,fit,'-')\n",
    "plt.title('Gaussion PDF')\n",
    "plt.xlabel('X with mean 0 and std 1')\n",
    "plt.ylabel('Probability')\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The PDF for the normal distribution is given by a rather formidable looking expression:\n",
    "<div style=\"font-size: 150%;\"> \n",
    "\\begin{equation}\n",
    "f(x, \\mu, \\sigma) = \\frac{1}{\\sigma\\sqrt{2\\pi}} e^{ -\\frac{(x-\\mu)^2}{2\\sigma^2} }\n",
    "\\end{equation}\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take a deep breath and relax though. \n",
    "\n",
    "All we need to do is calculate the mean $\\mu$, standard deviation $\\sigma$ and plug them together with our value $x$ into the equation in order to get the probability that our quantity lies within a small region of $x$ for each of the class labels."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Steps for calculating the classification for naive Bayes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each class:\n",
    "\n",
    "1. Calculate the probability for each of the features and then calculate their product - this represents the likelihood component of the Bayes Theorem.\n",
    "\n",
    "2. Multiply the likelihood by the prior for a given class (the proportion of all samples belonging to this class that make up the training dataset) - to give us the posterior probability that we are finally looking for.\n",
    "\n",
    "3. The class with the largest posterior probability is the class that wins the classification.\n",
    "\n",
    "4. Divide the posterior of each class with the sum of all posteriors in order to normalize and to finish up with a more interpretable probability."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** write a function to calculate the PDF for a normal distribution as below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pdf_gaussian(x, mean, std):\n",
    "    #YOUR CODE HERE\n",
    "    \n",
    "    return \n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = 66 \n",
    "mean = 73\n",
    "std = 6.2\n",
    "pdf_gaussian(x, mean, std)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given the wine dataset we saw previously: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\n",
    "    '../datasets/wine_data.csv',\n",
    "     usecols=[0,6,7]\n",
    "    )\n",
    "print(df.columns)\n",
    "df.columns=['Class','Magnesium','Flavanoids']\n",
    "df['Class'].replace(3, 0, inplace=True)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Classifier Representation and Training\n",
    "\n",
    "We are now going to implement the Naive Bayes classifier algorithm, and we are going to create a structure that stores and represents all the parameters that we need for this classifier.\n",
    "\n",
    "Fortunately, we do not need very much for Naive Bayes. We need:\n",
    "\n",
    "1. For each feature, the mean value for every class (**likelihood component**)\n",
    "2. For each feature, the standard deviation for every class (**likelihood component**)\n",
    "3. The probability for each class (**prior**)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "**Exercise:** Write a function takes a data frame and a class label and returns a dictionary containing the keys for each of the feature names in the dataset with an associated list having 2 elements, where the first is the mean and the second the standard deviation of the given feature:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calculate_class_statistics(df, class_label):\n",
    "    class_distr = {}\n",
    "    feature_types = []\n",
    "    \n",
    "    feature_types =  df.columns[1:]\n",
    "    means = (df[feature_types][df.Class == class_label]).mean()\n",
    "    stds = (df[feature_types][df.Class == class_label]).std()\n",
    "    \n",
    "    #YOUR CODE HERE\n",
    "\n",
    "       \n",
    "    return class_distr\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "calculate_class_statistics(df, 1)\n",
    "#SHOULD RETURN:\n",
    "# {'Magnesium': [2.8401694915254234, 0.3389613523154669],\n",
    "# 'Flavanoids': [2.982372881355932, 0.3974936086367577]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Write a function to train a NB classifier that takes in a data frame and returns a dictionary containing the following:\n"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "{'0': \n",
    "  {'Flavanoids': [0.78145833333333314, 0.29350406560186298],\n",
    "  'Magnesium': [1.6787500000000002, 0.35697085523801758],\n",
    "  'Prior': 0.2696629213483146},\n",
    " '1': \n",
    "  {'Flavanoids': [2.982372881355932, 0.3974936086367577],\n",
    "  'Magnesium': [2.8401694915254234, 0.33896135231546692],\n",
    "  'Prior': 0.33146067415730335},\n",
    " '2': \n",
    "  {'Flavanoids': [2.0808450704225359, 0.70570075908150276],\n",
    "  'Magnesium': [2.2588732394366198, 0.54536108430437336],\n",
    "  'Prior': 0.398876404494382}}\n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "where the key is the class label and the value is a dictionary, containing means and standard deviations for each feature for each class, as well as a key for the prior and its associated value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_naive_bayes(df):\n",
    "    NB_classifier = {}\n",
    "    class_labels = df.Class.unique()\n",
    "    \n",
    "    #YOUR CODE HERE\n",
    "\n",
    "    \n",
    "    return NB_classifier\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "NB_classifer = train_naive_bayes(df)\n",
    "NB_classifer\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** We are now going to move to classification. Write a function that takes as input a dictionary above as well as a series object to classify and returns the Bayes probability for this class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calculate_NB_probability_single_class(nb_classifier, series_x):\n",
    "    likelihood = 1.0 \n",
    "    \n",
    "    #YOUR CODE HERE\n",
    "\n",
    "    \n",
    "    return posterior\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "series_x = pd.Series([2.0, 2.3], index=['Flavanoids', 'Magnesium'])\n",
    "\n",
    "calculate_NB_probability_single_class(NB_classifer[str(1)], series_x)\n",
    "#SHOULD RETURN 0.0051879593513192053\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Write a function that classifies a series object into a class and returns its probability:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def classify_naive_bayes(NB_classifer, series_x):\n",
    "    classification = -1\n",
    "    classification_probabilities = []\n",
    "    prob = 0.0\n",
    "    total_ptobabilities = 0.0\n",
    "    \n",
    "    #YOUR CODE HERE\n",
    "    for i in NB_classifer:\n",
    "        \n",
    "    \n",
    "    #select winning class\n",
    "    for i in range(len(classification_probabilities)):\n",
    "        \n",
    "    \n",
    "    #normalize probability\n",
    "    return \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "series_x = pd.Series([4.0, 2.3], index=['Flavanoids', 'Magnesium'])\n",
    "\n",
    "winning_class, prob = classify_naive_bayes(NB_classifer, series_x)\n",
    "winning_class, prob\n",
    "\n",
    "#SHOULD RETURN ('1', 0.50456450520692542)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Congratulations you have now fully implemented, trained and deployed your first classifier!**\n",
    "\n",
    "**Exercise:** Write a function that takes in at NB classifier and a dataset and classifies each sample in the dataset. It creates two new columns on the dataset called 'Classification' and 'Probability' of the classification and returns the data frame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def classify_dataset(NB_classifer, df):\n",
    "    res = []\n",
    "    #YOUR CODE HERE\n",
    "    for i in range(len(df)):\n",
    "        \n",
    "    \n",
    "    return pd.concat([df, res_df], axis=1)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df_result = classify_dataset(NB_classifer, df)\n",
    "df_result.head(100)\n",
    "\n"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "#SHOULD RETURN SOMETHING LIKE:\n",
    "   Class \tMagnesium \tFlavanoids \tClassification \tProbability\n",
    "0 \t1          2.80 \t3.06                1       0.908310\n",
    "1 \t1          2.65 \t2.76                1       0.780886\n",
    "2 \t1          2.80 \t3.24                1       0.923342\n",
    "3 \t1          3.85 \t3.49                1       0.865404\n",
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comparing Naive Bayes to k-NN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">  Naive Bayes is a linear classifier, while k-NN is not. The\n",
    "curse of dimensionality and large feature sets are a problem for k-NN,\n",
    "while Naive Bayes performs well. k-NN requires no training (just load\n",
    "in the dataset), whereas Naive Bayes does. Both are examples of supervised\n",
    "learning (the data comes labeled). \n",
    "\n",
    "Schutt, R., & O'Neil, C. (2013). Doing Data Science: Straight Talk from the Frontline. \" O'Reilly Media, Inc.\".\n",
    "\n",
    "\n",
    "kNN is in many ways a special case of a supervised machine learning algorithm. It is unique in that the data itself is the model and no training of a classifier takes place explicitly. **kNN is particularly susceptible to deteriorating accuracy** if there are meaningless features in the dataset. Hence, feature analysis should always be performed together with dimensionality reduction if using kNN.\n",
    "\n",
    "**Naive Bayes is more robust against outliers as well as un-informative features** than kNN. However, one has to be careful with Naive Bayes because it is naive in terms of assuming independence between features, which is almost never true in reality. Therefore **Naive Bayes is vulnerable when this assumption is strongly violated and when there is a large presence of highly correlated and redundant features**, the training will bias the final result towards those features, and the final probabilities are unlikely to be accurate for interpretation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** \n",
    "\n",
    "Go to https://archive.ics.uci.edu/ml/index.php and select 3 machine learning datasets. \n",
    "\n",
    "Load them into the notebook. Prepare them for classification.\n",
    "\n",
    "Perform 10-fold stratified-cross-validation using the built in scikit-learn Naive Bayes.\n",
    "\n",
    "Evaluate the generalisation using several metrics.\n",
    "\n",
    "Perform feature selection and investigate if the generalisation has improved."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:teaching]",
   "language": "python",
   "name": "conda-env-teaching-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
